For large CDs this makes it faster to identify
changes from the log output.

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>

This is less error-prone: if we treat `node ==
nil` generally as success, we may miss persisting
a pod state transition in edge cases and for
edge-case state transitions after the initial
NotReady -> Ready transition.

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>

This reduces the amount of log volume on the
default log level for large CDs.

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>

This was meant to be three seconds, not 3000 seconds.

This is a node-local retry and we can easily
afford not backing off towards O(1 min) or
further.

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>

Change upper bound from 1000 s (17 minutes) to something much less.

For formation of a larger ComputeDomain (N nodes),
many writers desire updating the same API server
object.

The overall work that has to be done by the API
server scales linearly with N: a certain number of
updates (at least N, in the best case) is
required.

Hence, in the best case (perfect serialization, no
conflicts) the time it takes overall to get all
updates in (the overall ideal convergence time C)
is scaling with N, linearly.

In the worst case, when individual updates always
conflict with each other (are performed 'at the
same time' against the same reference state),
convergence is never achieved.

Without centralized coordination, backing off
individual retriers is a way to spread out updates
over time. The nature of the distribution of those
back-offs governs how long the actual convergence
time takes compared to the ideal case C.

The ideal case C is governed by the rate R at
which the central entity can process updates.

If we naively back off exponentially without a
sane upper bound then we don't homogenously spread
the update load over time, but try to get less and
less updates injected into the system per time, as
time progresses. The attempted update rate then
falls far below R (the possible update rate). That
makes convergence unnecessarily slow.

If we do not back off enough, an opposite effect
may occur because the global rate of retries
accumulating at the central point (API server) may
always exceed R, and hence thrash resources and
slow things down compared to the theoretical
update rate maximum (in case of perfectly
serialized updates).

Hence, there is a sweet spot between both extrema.
The positioning of that sweet spot strongly
depends on R.

Summary:

1) We do not want to back off individual retriers
   too far, otherwise we operate at an update rate
   lower than necessary and artificially slow down
   the convergence process.

2) We need to back off individual retriers enough
   to prevent thrashing from slowing us and others
   down. This is critical for making sure the
   convergence time scales linearly with N
   (instead of, say, O(N**2)).

This patch primarily takes care of (1).

For (2), in the future, we may want to further
increase that upper bound after a certain amount
of time (if e.g. a 5 second cap does not result to
overall convergence after e.g. 30 minutes, it may
be worth backing off further, to remove stress
from the API server).

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>

The client-go rate limiters such as
`ExponentialFailureRateLimiter` do not implement
jitter. In a user's environment, formation of a CD
across 144 nodes has shown that the absence of
jitter results in significant retry attempt
correlation across nodes -- even after ~10
retries, resulting in otherwise preventable
conflicts (and hence increased convergence time).

That effect can be diminished by adding jitter,
which should allow for

The JitterRL implementation provided by this patch
is a simple, custom implementation that I
validated with simulated errors and careful
placement of log messages.

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>

See issue NVIDIA#742.

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>

Generally, output looks cleaner with that.
Makes sorting lexicographically more meaningful
(otherwise 10 comes right after 1).

Should be OK even for upgraded systems as IMEX daemon
nodes config and /etc/hosts are written in tandem.

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>

Sort by DNS name (map value).

Signed-off-by: Dr. Jan-Philip Gehrcke <[email protected]>